Advances and Challenges for Scalable Provenance in Stream Processing Systems

While data provenance is a relatively well-studied topic in both the fields of databases and workflow systems, its support within stream processing systems presents a new set of challenges. Given the potentially high event rate of the input streams and the low processing latency requirements imposed by many streaming applications, capturing data provenance effectively in a stream processing system is extremely challenging. Regardless, emerging streaming applications call for data provenance support (e.g., healthcare analytics applications, financial applications). To illustrate this point, data provenance support has become an essential part of the Century stream processing infrastructure that we are building for supporting online healthcare analytics. At anytime, given an output data element (e.g., a medical alert) generated by the Century system, the system must be able to retrieve all the input and intermediate data elements that led to its generation. In this paper, we first describe the concrete requirements behind our initial implementation of Century’s provenance subsystem. We then analyze the strengths and limitations of our current solution, and propose a new provenance architecture to address some of these observed limitations. The paper also includes a discussion on a set of open challenges and issues that we must resolve.